Information processing apparatus, imaging device, information processing method, and computer program

ABSTRACT

An information processing apparatus includes a data processor configured to execute processing for selecting a plurality of frame pictures from moving-picture data and for displaying an array of the selected frame pictures. In selecting frame pictures to be displayed, the data processor preferentially selects frame pictures corresponding to highlight points as frame pictures to be displayed, using highlight point information that allows identification of picture portions corresponding to important scenes, the highlight point information being recorded in association with the moving-picture data.

CROSS REFERENCES TO RELATED APPLICATIONS

The present invention contains subject matter related to Japanese PatentApplication JP 2005-323364 filed in the Japanese Patent Office on Nov.8, 2005, the entire contents of which are incorporated herein byreference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to information processing apparatuses,imaging devices, information processing methods, and computer programs.More specifically, the present invention relates to an informationprocessing apparatus, an imaging device, an information processingmethod, and a computer program that allow selection of representativepictures from captured moving-picture data and display of an array ofthe selected representative pictures, such as a film-roll display.

2. Description of the Related Art

In a known method of presenting the content of a moving picture capturedby a video camera to a user in a readily recognizable manner, framepictures are selected from the moving picture, for example, at constantintervals, and the frame pictures are arranged in a temporal order sothat the frame pictures are displayed like a film roll (hereinafterreferred to as a “film-roll display”).

With the film-roll display, the user can recognize the content of themoving picture without playing back the moving picture. For example, bydisplaying a list of frame pictures selected from various moving-picturedata files captured in the past, the user can efficiently check capturedpictures. Also, it is possible to display representative picturesselected from picture data that is being played back, as a film-rolldisplay in a lower part of the screen, the representative pictures beingdisplayed as superposed on the picture that is being played back, sothat the user can select a still picture in the film-roll display tospecify a playback position. In an ordinary film-roll display, an arrayof thumbnail pictures (reduced pictures) is displayed. Such a method ofdisplaying pictures is described, for example, in Japanese UnexaminedPatent Application Publication No. 2-32473.

However, particularly in devices having only small screens, such asvideo cameras or digital cameras, it is not possible to display an arrayof a sufficient number of pictures. Thus, the number of picturesdisplayed in a film-roll display is small, so that it is often difficultfor the user to grasp the entirety of a moving picture. Also, whenrepresentative still pictures are displayed as a film-roll display assuperposed on a moving picture that is being played back, as the lengthof the recorded moving picture becomes longer, it becomes more difficultto display an array of suitable representative pictures. Thus, itbecomes difficult for the user to select a suitable playback position.

SUMMARY OF THE INVENTION

It is desired that an information processing apparatus, an imagingdevice, an information processing method, and a computer program beprovided so that when displaying an array of representative picturesselected from captured moving-picture data, for example, in a film-rolldisplay, instead of displaying pictures that are selected at constanttime intervals, pictures considered as important scenes in a movingpicture are efficiently selected and displayed, thereby facilitatingrecognition of the content of the moving picture data and allowingspecification of a playback position corresponding to an important scenethat is presumably highly demanded by a user.

According to an embodiment of the present invention, there is providedan information processing apparatus including a data processorconfigured to execute processing for selecting a plurality of framepictures from moving-picture data and for displaying an array of theselected frame pictures. In selecting frame pictures to be displayed,the data processor preferentially selects frame pictures correspondingto highlight points as frame pictures to be displayed, using highlightpoint information that allows identification of picture portionscorresponding to important scenes, the highlight point information beingrecorded in association with the moving-picture data.

In the information processing apparatus, in selecting frame pictures tobe displayed, the data processor may calculate reference timescorresponding to a plurality of display frames that are set along atemporal sequence, and for each of the reference times, the dataprocessor may preferentially select a frame picture corresponding to ahighlight point having a recording time nearest to the reference time asa frame picture to be displayed.

Also, in the information processing apparatus, in selecting framepictures to be displayed, the data processor may calculate referencetimes corresponding to a plurality of display frames that are set alonga temporal sequence, and for each of the reference times, the dataprocessor may preferentially select a frame picture corresponding to ahighlight point having a recording time nearest to the reference time asa frame picture to be displayed, and when no frame picture correspondingto the highlight point exists in a predetermined threshold time range,the data processor may select a frame picture corresponding to thereference time as a frame picture to be displayed.

Also, in the information processing apparatus, the data processor mayobtain recording times of important scenes from highlight pointinformation including the recording times of the important scenes, andfor each of the recording times of the important scenes, the dataprocessor may select a frame picture corresponding to a highlight pointhaving the recording time of the important scene as a frame picture tobe displayed when the following conditional expression is satisfied:x−(Δx/2)≦tj<x+(Δx/2)where tj denotes the recording time of the important scene, x denotes areference time corresponding to a display frame associated with therecording time of the important scene among a plurality of displayframes that are set along a temporal sequence, and Δx denotes a timeinterval of the display frames.

According to another embodiment of the present invention, there isprovided an imaging device including a recording data processorconfigured to execute processing for selecting picture portionscorresponding to important scenes from captured picture data on thebasis of input information that is input during a picture taking period,for generating highlight point information that allows identification ofthe selected picture portions corresponding to the important scenes, andfor recording the highlight point information on a recording medium asinformation associated with the captured picture data; and a playbackdata processor configured to execute processing for selecting aplurality of frame pictures from moving-picture data and for displayingan array of the selected frame pictures. In selecting frame pictures tobe displayed, the playback data processor preferentially selects framepictures corresponding to highlight points as frame pictures to bedisplayed, using the highlight point information recorded in associationwith the moving-picture data.

In the imaging device, the recording data processor may receive, as theinput information, at least one of zoom operation information, detectioninformation of an acceleration sensor, audio signal level information,GPS information, clock information, and still-picture takinginformation, identifies the picture portions corresponding to theimportant scenes on the basis of the input information, and generatesthe highlight point information.

According to another embodiment of the present invention, there isprovided an information processing method for executing processing fordisplaying a picture by an information processing apparatus. Theinformation processing method includes the step of executing dataprocessing for selecting a plurality of frame pictures frommoving-picture data and for displaying an array of the selected framepictures, the data processing being executed by a data processor. Inselecting frame pictures to be displayed, frame pictures correspondingto highlight points are preferentially selected as frame pictures to bedisplayed, using highlight point information that allows identificationof picture portions corresponding to important scenes, the highlightpoint information being recorded in association with the moving-picturedata.

In the information processing method, in selecting frame pictures to bedisplayed, reference times corresponding to a plurality of displayframes that are set along a temporal sequence may be calculated, and foreach of the reference times, a frame picture corresponding to ahighlight point having a recording time nearest to the reference timemay be preferentially selected as a frame picture to be displayed.

Also, in the information processing method, in selecting frame picturesto be displayed, reference times corresponding to a plurality of displayframes that are set along a temporal sequence may be calculated, and foreach of the reference times, a frame picture corresponding to ahighlight point having a recording time nearest to the reference timemay be preferentially selected as a frame picture to be displayed, andwhen no frame picture corresponding to the highlight point exists in apredetermined threshold time range, a frame picture corresponding to thereference time may be selected as a frame picture to be displayed.

Also, in the information processing method, recording times of importantscenes may be obtained from highlight point information including therecording times of the important scenes, and for each of the recordingtimes of the important scenes, a frame picture corresponding to ahighlight point having the recording time of the important scene may beselected as a frame picture to be displayed when the followingconditional expression is satisfied:x−(Δx/2)≦tj<x+(Δx/2)where tj denotes the recording time of the important scene, x denotes areference time corresponding to a display frame associated with therecording time of the important scene among a plurality of displayframes that are set along a temporal sequence, and Δx denotes a timeinterval of the display frames.

According to another embodiment of the present invention, there isprovided an information processing method executed by an imaging device.The information processing method includes the steps of executingrecording data processing for selecting picture portions correspondingto important scenes from captured picture data on the basis of inputinformation that is input during a picture taking period, for generatinghighlight point information that allows identification of the selectedpicture portions corresponding to the important scenes, and forrecording the highlight point information on a recording medium asinformation associated with the captured picture data, the recordingdata processing being executed by a recording data processor; andexecuting playback data processing for selecting a plurality of framepictures from moving-picture data and for displaying an array of theselected frame pictures, the playback data processing being executed bya playback data processor. In selecting frame pictures to be displayed,frame pictures corresponding to highlight points are preferentiallyselected as frame pictures to be displayed, using the highlight pointinformation that allows identification of the picture portionscorresponding to the important scenes, the highlight point informationbeing recorded in association with the moving-picture data.

In the information processing method, as the input information, at leastone of zoom operation information, detection information of anacceleration sensor, audio signal level information, GPS information,clock information, and still-picture taking information is received, thepicture portions corresponding to the important scenes are identified onthe basis of the input information, and the highlight point informationmay be generated.

According to another embodiment of the present invention, there isprovided a computer program for allowing an information processingapparatus to execute processing for displaying an image. The computerprogram includes the step of executing data processing for selecting aplurality of frame pictures from moving-picture data and for displayingan array of the selected frame pictures, the data processing beingexecuted by a data processor. In selecting frame pictures to bedisplayed, frame pictures corresponding to highlight points arepreferentially selected as frame pictures to be displayed, usinghighlight point information that allows identification of pictureportions corresponding to important scenes, the highlight pointinformation being recorded in association with the moving-picture data.

According to another embodiment of the present invention, there isprovided a computer program for allowing an imaging device to executeinformation processing. The computer program includes the steps ofexecuting recording data processing for selecting picture portionscorresponding to important scenes from captured picture data on thebasis of input information that is input during a picture taking period,for generating highlight point information that allows identification ofthe selected picture portions corresponding to the important scenes, andfor recording the highlight point information on a recording medium asinformation associated with the captured picture data, the recordingdata processing being executed by a recording data processor; andexecuting playback data processing for selecting a plurality of framepictures from moving-picture data and displaying an array of theselected frame pictures, the playback data processing being executed bya playback data processor. In selecting frame pictures to be displayed,frame pictures corresponding to highlight points are preferentiallyselected as frame pictures to be displayed, using the highlight pointinformation that allows identification of the picture portionscorresponding to the important scenes, the highlight point informationbeing recorded in association with the moving-picture data.

Each of the computer programs according to these embodiments can beprovided using a storage medium or communication medium for providingthe computer program in a computer-readable form to, for example, ageneral-purpose computer system that is capable of executing variousprogram codes, e.g., a storage medium such as a CD, an FD, or an MO, ora communication medium such as a network. By providing the computerprogram in a computer-readable form, processing can be executed on thecomputer system according to the computer program.

Other objects, features, and advantages of the present invention willbecome apparent from the embodiments of the present invention describedbelow in more detail with reference to the attached drawings. In thisspecification, a system refers to a logical combination of a pluralityof apparatuses, regardless of whether the apparatuses reside within asingle case.

According to an embodiment of the present invention, when a plurality offrame pictures are selected from moving-picture data captured by, forexample, a digital video camera and an array of the selected framepictures is displayed, for example, a film-roll display, in selectingframe pictures to be displayed, frame pictures corresponding tohighlight points are preferentially selected as frame pictures to bedisplayed, using highlight point information that allows identificationof picture portions corresponding to important scenes, the highlightpoint information being recorded in association with the moving-picturedata. Thus, many pictures presented in the film-roll display representimportant scenes. Therefore, a user can efficiently recognize thecontent of the moving picture and select scenes to be played back.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the configuration of an imaging deviceaccording to an embodiment of the present invention;

FIG. 2 is a flowchart showing the sequence of a highlight-pointextracting process based on zoom operation information;

FIGS. 3A and 3B are diagrams showing an example of the highlight-pointextracting process based on zoom operation information and an exampledata structure of highlight point information based on zoom operationinformation;

FIG. 4 is a flowchart showing the sequence of a highlight-pointextracting process based on acceleration sensor information;

FIGS. 5A and 5B are diagrams showing an example of the highlight-pointextracting process based on acceleration sensor information and anexample data structure of highlight point information based onacceleration sensor information;

FIG. 6 is a diagram for explaining a highlight-point extracting processbased on audio information;

FIG. 7 is a flowchart showing the sequence of the highlight-pointextracting process based on audio information;

FIGS. 8A and 8B are diagrams showing an example of the highlight-pointextracting process based on audio information and an example datastructure of highlight point information based on audio information;

FIG. 9 is a flowchart showing the sequence of a highlight-pointextracting process based on GPS information;

FIGS. 10A to 10C are diagrams showing an example of the highlight-pointextracting process based on GPS information and an example datastructure of highlight point information;

FIG. 11 is a flowchart showing the sequence of a highlight-pointextracting process based on clock information;

FIGS. 12A and 12B are diagrams showing an example of the highlight-pointextracting process based on clock information and an example datastructure of highlight point information based on clock information;

FIG. 13 is a flowchart showing the sequence of a highlight-pointextracting process based on still-picture taking information;

FIGS. 14A and 14B are diagrams showing an example of the highlight-pointinformation based on still-picture taking information and an exampledata structure of highlight point information based on still-picturetaking information;

FIGS. 15A to 15F are diagrams showing pieces of highlight pointinformation obtained by highlight-point extracting processes based onvarious types of information;

FIG. 16 is a flowchart showing the sequence of a highlight-pointselecting process executed to play back highlight scenes;

FIG. 17 is a diagram for explaining an example of playback of highlightscenes;

FIG. 18 is a diagram for explaining an example of film-roll display;

FIG. 19 is a diagram for explaining an example of film-roll display; and

FIG. 20 is a flowchart showing the sequence of a film-roll displayingprocess.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Now, information processing apparatuses, imaging devices, informationprocessing methods, and computer programs according to embodiments ofthe present invention will be described in detail with reference to thedrawings.

First, an example configuration of an imaging device (digital camera) asan information processing apparatus according to an embodiment of thepresent invention will be described with reference to FIG. 1.

Referring to FIG. 1, the imaging device includes a lens 101, an imager102 implemented by a solid state imaging device (e.g., CCD) or the likethat converts optical signals input via the lens 101 into electricsignals, an A/D converter 103 that converts the analog electric signalsinto digital signals, a camera signal processor 104 that executes signalprocessing on pictures on the basis of captured video data, a microphone105 that receives input of audio information, an A/D converter 106 thatconverts the analog audio signals output from the microphone 105 intodigital signals, and an audio signal processor that executes signalprocessing on sounds on the basis of captured audio data. The audiosignal processor 107 inputs audio signal level information to amicroprocessor 111. The microprocessor 111 extracts highlight points(pieces of information representing positions of important scenes), forexample, on the basis of the audio signal level information input fromthe audio signal processor 107.

The microprocessor 111 functions as a recording data processor thatobtains various types of input information during an imaging period,that selects portions corresponding to important scenes from capturedvideo data, that generates highlight point information that allowsidentification of the portions corresponding to the important scenes,selected on the basis of the input information, and that records thehighlight point information on a recording medium 112 as informationassociated with the captured video data. The extraction of highlightpoints will be described later in more detail.

The microprocessor 111 also functions as a playback data processor thatplays back and displays captured pictures, that selects pictures to bedisplayed in a film roll display, and so forth. The film roll displaywill be described later in more detail.

The video and audio digital data generated by the camera signalprocessor 104 and the audio signal processor 107 are stored in a memory120. A video encoder 121 and an audio encoder 122 execute encoding toconvert the input video signals and audio signals into data forrecording. A video decoder 123 and an audio decoder 124 execute decodingof video and audio signals read from the recording medium 112, forexample, when data is read and played back from the recording medium 112having captured data recorded thereon. The memory 120 temporarily storesdata to be encoded or decoded.

The microprocessor 111 controls processing executed in the imagingdevice. Also, the microprocessor 111 receives information input by auser from input devices 108, such as various switches and buttons.Furthermore, the microprocessor 111 receives sensor detectioninformation from an acceleration sensor 109, receives audio signal levelinformation from the audio signal processor 107, receives positioninformation obtained by a GPS module 110 from GPS satellites, andreceives date and time information from a clock 131. Furthermore, themicroprocessor 111 controls recording or playback of data on or from therecording medium 112.

The input devices 108 are buttons, switches, or the like that are usedwhen a user operates the imaging device. More specifically, the inputdevices 108 include, for example, a switch for a zoom operation, a keyfor taking a moving picture, a shutter key for taking a still picture,and a touch panel. Information that is input via the input devices 108is transmitted to the microprocessor 111, and the microprocessor 111exercises control and extracts highlight points on the basis of theinformation corresponding to user's operations.

The acceleration sensor 109 detects movement of the imaging device, andinputs information representing the result of detection to themicroprocessor 111. The GPS module 110 receives data from GPSsatellites, and inputs the data to the microprocessor 111. On the basisof this information, the microprocessor 111 can obtain the latitude andlongitude of the current position. The clock 131 inputs informationrepresenting the current date and time to the microprocessor 111. Themicroprocessor 111 extracts highlight points on the basis of thesepieces of information input from the acceleration sensor 109, the GPSmodule 110, and the clock 131.

The recording medium 112 is, for example, a non-volatile memory or ahard disk, and the recording medium 112 stores captured video data. Adisplay controller 125 controls monitor display of captured video data,or display of data already captured and recorded on the recording medium112 and decoded by the video decoder 123. Data generated by the displaycontroller 125 is converted into analog signals by a D/A converter 126,and the analog signals are output to a display 127, such as an LCDpanel.

An audio signal processor 128 receives input of audio signals decoded bythe audio decoder 124. The decoded audio signals are converted intoanalog signals by a D/A converter 129, and the analog signals are outputto a speaker 130.

The microprocessor 111, which functions as a data processor in theimaging device according to this embodiment, receives input of thefollowing pieces of information:

(1) Information regarding user's operations of the input devices 108

(2) Information regarding results of detection by the accelerationsensor 109

(3) Audio signal level information supplied from the audio signalprocessor 107

(4) Position information supplied from the GPS module 110

(5) Date and time information supplied from the clock 131

On the basis of these pieces of input information, the microprocessor111 extracts highlight points (information representing positions ofimportant scenes). The microprocessor 111 receives input of these piecesof information and extracts highlight points on the basis of the inputinformation during an imaging period. The extraction of highlight pointswill be described below in detail.

First, highlight points will be described. A highlight point is a pieceof information that allows identification of a scene that is presumablyimportant in a recorded moving picture. For example, a highlight pointis represented in terms of a time elapsed from the beginning ofrecording of a moving picture (hereinafter referred to as a “recordingtime”). That is, a highlight point is a piece of informationrepresenting a temporal position of a frame including an important scenerelative to the beginning of imaging.

In the imaging device according to this embodiment, for example, thefollowing points are detected as highlight points:

A point at which the user performed a zoom operation while taking apicture (represented in terms of a time from the beginning of imaging);

A point at which the user moved the camera (performed a pan or tiltoperation) while taking a picture;

A point at which the level of sound input from the microphone reached apeak;

A point at which information representing the latitude or longitudechanged, the information being obtained from GPS satellites;

A point corresponding to a specific change in the current time (e.g., achange of the day or a change of the year); and

A point at which the user recorded a still picture while recording amoving picture.

The imaging device according to this embodiment detects these highlightpoints while recording a moving picture, and records the highlightpoints as highlight point information associated with captured videodata. With the highlight points recorded as described above, when thecaptured data is played back, it is possible to selectively play backonly portions corresponding to the proximities of the highlight points.Thus, it is possible to selectively play back highlight scenesimmediately after imaging, without executing particular post processingon captured data, such as image analysis.

Now, the extraction of highlight points by the imaging device accordingto this embodiment will be described in detail, regarding the followingpieces of detected information in order:

(1) Zoom operation information associated with user's operations of theinput devices 108

(2) Sensor detection information supplied from the acceleration sensor109

(3) Audio signal level information supplied from the audio signalprocessor 107

(4) Position information supplied from the GPS module 110

(5) Date and time information supplied from the clock 131

(6) Still-picture taking information associated with user's operationsof the input devices 108

(1) Zoom Operation Information Associated with User's Operations of theInput Devices 108

First, a highlight-point extracting process that is executed by themicroprocessor 111 when zoom operation information associated with auser's operation of the input devices 108 is input to the microprocessor111 will be described in detail with reference to FIG. 2 and FIGS. 3Aand 3B.

FIG. 2 is a flowchart showing a sequence of the highlight-pointextracting process that is executed during imaging when zoom operationassociated with a user's operation of the input devices 108 is input tothe microprocessor 111.

While taking a moving picture, a user operates a zoom key, which is oneof the input devices 108, to change the zoom ratio. The user's operationis reported from the input devices 108 to the microprocessor 111, andthe microprocessor 111 moves the lens 101 to change the zoom ratio. Whenthe user stops the zoom operation, for example, by releasing the zoomkey, the microprocessor 111 stops moving the lens 101 and thereby stopsthe zoom operation.

Now, processing in the individual steps of the processing flow shown inFIG. 2 will be described. When the microprocessor 111 has detected thestart of a zoom operation in step S101, the process proceeds to stepS102, in which information representing the magnification factor at thestart of the zoom operation is obtained.

Then, when the end of the zoom operation is detected in step S103, instep S104, information representing the magnification factor at the endof the zoom operation is obtained. Then, in step S105, a data set of(current recording time, amount of zoom change) is stored on the memory120.

The amount of zoom change herein refers to, for example, an amount ofchange in the zoom ratio, and is calculated on the basis of thedifference between the zoom ratio at the end of the zoom operation andthe zoom ratio at the start of the zoom operation.

Then, in step S106, it is detected whether video recording has beenfinished. When video recording has not been finished, the processreturns to step S101 and is repeated therefrom. Through this repetition,data sets representing the end times of individual zoom operationsexecuted during video recording and the amounts of zoom change in theindividual zoom operations, i.e., data sets of (current recording time,amount of zoom change), are sequentially stored on the memory 120.

When it is detected in step S106 that video recording has been finished,in step S107, the data sets of (current recording time, amount of zoomchange) stored on the memory 120 are recorded in association withcaptured data, for example, on the recording medium 112, as highlightinformation associated with the captured data.

The user typically performs a zoom operation, for example, when the userwishes to zoom in to a subject of imaging or to zoom out to have a widerview. Thus, it can be reasonably presumed that a recording timeimmediately after a zoom operation is likely to correspond to animportant scene, so that the recording time can be considered as ahighlight point.

When data sets of (current recording time, amount of zoom change) storedon the memory 120 are recorded on the recording medium 112 as highlightpoint information associated with captured data at the end of recordingof a moving picture, it is possible to record all the information storedon the memory 120 as highlight point information. Alternatively, it ispossible to select a predetermined number of data sets having largeamounts of zoom change from the information stored on the memory 120 andto record only the selected data sets on the recording medium 112 ashighlight point information together with moving-picture data.

Various schemes may be employed for the recording of highlight pointinformation. For example, an area for recording highlight points may beprovided in a part of a moving-picture file, or a database file may beprovided separately from a moving-picture file so that highlight pointscan be recorded in the database file.

Now, a specific example of the highlight-point extracting process basedon zoom operation information and a specific example of data ofhighlight point information based on zoom operation information will bedescribed with reference to FIGS. 3A and 3B. FIG. 3A is a diagramshowing a history of zoom operations in relation to time elapsed fromthe beginning (Start) to end (Stop) of taking of a moving picture. FIG.3B is a diagram showing an example of data of highlight pointinformation associated with captured video data, which is recorded onthe recording medium 112 when the zoom operations shown in FIG. 3A areexecuted.

In the example shown in FIG. 3A, three zoom operations are executedbetween the beginning and end of imaging. The first zoom operation is azoom operation in a direction of wide (W) to telephoto (T), i.e., a zoomin toward a subject of imaging, and the first zoom operation is ended at0 minutes and 45 seconds after the beginning of imaging. The end time ofthe zoom operation is recorded as a highlight point. The data recordedcorresponds to the first entry data of highlight point information shownin FIG. 3B. That is, a data set of the end time of the zoom operationand the amount of zoom change, i.e., [0:45] and [3.0], is recorded as afirst piece of highlight point information.

Similarly, as second and third pieces of highlight point information,data sets of end time and amount of zoom change for the individual zoomoperations are recorded. In the example shown in FIG. 3B, the amount ofzoom change in the direction of wide (W) to telephoto (T) is representedas a positive value, and the amount of change in the direction oftelephoto (T) to wide (W) is represented as a negative value.

(2) Sensor Detection Information Supplied from the Acceleration Sensor109

Next, a highlight-point executing process that is executed by themicroprocessor 111 when information is input from the accelerationsensor 109 to the microprocessor 111 will be described with reference toFIG. 4 and FIGS. 5A and 5B. In this process, a pan or tilt operationexecuted during imaging is detected by the microprocessor 111 on thebasis of information input from the acceleration sensor 109, and ahighlight point is recorded on the basis of this detection.

When the user moves the camera while taking a moving picture, theacceleration sensor 109 detects the movement and reports the movement tothe microprocessor 111. When the microprocessor 111 determines that theimaging device is continuously exhibiting movement in the same directionover a certain period on the basis of the information input from theacceleration sensor 109, the microprocessor 111 determines that the useris moving the camera in a certain direction (i.e., executing a pan ortilt operation), and records the current point as a highlight point.

Now, processing executed in the individual steps of the processing flowshown in FIG. 4 will be described. In step S201, as initial setting, themicroprocessor 111 resets the amount of continuous movement stored inthe memory 120 to [0]. Then, in step S202, a value representing a resultof detection is obtained from the acceleration sensor 109. Then, in stepS203, the microprocessor 111 checks whether the imaging device is in astill state or a moving state.

When it is determined that the imaging device is in a moving state, theprocess proceeds to step S211, in which it is determined whether thedirection of movement is the same as the direction of movement indicatedby information input from the acceleration sensor 109 last time. Theinput of information from the acceleration sensor 109 is repeatedsuccessively. When it is determined that the direction of movement isthe same as the direction of movement indicated by information inputfrom the acceleration sensor 109 last time, it is determined likely thata pan or tilt operation is being executed. Then, the amount of movementis calculated on the basis of the information input from theacceleration sensor 109, and the amount of continuous movement stored inthe memory 120 is updated. The amount of movement can be obtained bycalculating a velocity from acceleration and temporally integrating thevelocity.

When it is determined in step S211 that the direction of movement is notthe same as the direction of movement indicated by information inputfrom the acceleration sensor 109 last time, it is determined not likelythat a pan or tilt operation is being executed. Then, the processproceeds to step S205, in which the amount of continuous movement storedin the memory 120 is reset.

When it is determined in step S203 on the basis of information inputfrom the acceleration sensor 109 that the imaging device is in a stillstate, the process proceeds to step S204. In step S204, the valuerepresenting the amount of continuous movement, stored in the memory120, is referred to, and it is determined whether the amount of movementis greater than a predetermined threshold distance. When the amount ofmovement is greater than the predetermined threshold distance, it isdetermined that a pan or tilt operation has been executed. Then, in stepS213, a data set of (current recording time, amount of movement) isstored in the memory 120.

When it is determined in step S204 that the value representing theamount of continuous movement, stored in the memory 120, is not greaterthan the predetermined threshold distance, it is determined that a panor tilt operation has not been executed. Then, in step S205, the valuerepresenting the amount of continuous movement, stored in the memory120, is reset to [0].

In step S206, it is determined whether video recording has beenfinished. When video recording has not been finished, the processreturns to step S202 and is repeated therefrom. Through the repetition,data sets of end time of each pan or tilt operation executed duringimage recording and the amount and direction of movement in the pan ortilt operation, i.e., data sets of (current recording time, amount anddirection of movement), are sequentially stored on the memory 120.

When it is detected in step S206 that video recording has been finished,in step S207, the data sets of (current recording time, amount anddirection of movement) stored in the memory 120 are recorded, forexample, on the recording medium 112 in association with captured dataas highlight point information associated with the captured data.

The user typically performs a pan or tilt operation, for example, whenthe user wishes changes a subject of interest for imaging. Thus, a pointimmediately after a pan or tilt operation can be reasonably presumed ascorresponding to an important scene, so that the point can be consideredas a highlight point. For example, when a pan or tilt operation has beenfinished and the camera has become still, the current recording time andthe amount and direction of movement caused by the pan or tilt operationare stored on the memory 120 as highlight information. When therecording of the moving picture has been finished, all the highlightpoint information that has been stored, or a predetermined number ofsets of highlight point information with larger amounts of movementcaused by the pan or tilt operations is recorded on the recording medium112 together with moving-picture data. Similarly to recording ofhighlight points associated with zoom operations described earlier,various schemes may be employed for the recording of highlight points.

Now, a specific example of the highlight-point extracting process basedon pan or tilt information and a specific example of data of highlightpoint information based on pan or tilt information will be describedwith reference to FIGS. 5A and 5B. FIG. 5A is a diagram showing ahistory of pan and tilt operations in relation to time elapsed from thebeginning (Start) to end (Stop) of taking of a moving picture. FIG. 5Bis a diagram showing an example of data of highlight informationassociated with captured video data, which is recorded on the recordingmedium 112 when the pan and tilt operations shown in FIG. 5A areexecuted.

In the example shown in FIG. 5A, three pan or tilt operations areexecuted between the beginning and end of imaging. The first pan or tiltoperation is ended at 0 minutes and 52 seconds after the beginning ofimaging. The end time is recorded as a highlight point. The datarecorded corresponds to the first entry data of highlight pointinformation shown in FIG. 5B. That is, a data set of the end time of thepan or tilt operation and the amount and direction of movement, i.e., adata set of [0:52] and [5.0], is recorded as a first piece of highlightpoint information.

Similarly, as the second and third pieces of highlight pointinformation, data sets of the end time and the amount and direction ofmovement of the individual pan or tilt operations are recorded. In theexample shown in FIG. 5B, the amount of movement in the direction ofleft to right and in the direction of upper to lower is represented by apositive number, and the amount of movement in the opposite directionsare represented by a negative number.

In the example described above, the amount of movement associated with apan or tilt operation is obtained by calculating a velocity from anacceleration input from the acceleration sensor 109 and temporallyintegrating the velocity. However, even in a device not including anacceleration sensor, it is possible to obtain information regardingcamera movement by comparing pictures of adjacent frames included incaptured video data. Thus, the configuration may be such that themicroprocessor 111 determines whether a pan or tilt operation has beenexecuted on the basis of captured video data.

(3) Audio Signal Level Information Supplied from the Audio SignalProcessor 107

Next, a highlight-point extracting process that is executed by themicroprocessor 111 when audio signal level information supplied from theaudio signal processor 107 is input to the microprocessor 111 will bedescribed in detail with reference to FIG. 6., FIG. 7, and FIGS. 8A and8B. In this process, audio signal level information based on audio datacaptured by the microphone 105 during imaging is input to themicroprocessor 111, and the microprocessor 111 analyzes change in theaudio level on the basis of the input information and records highlightpoints on the basis of the result of the analysis.

Sound is input via the microphone 105 while taking a moving picture, andaudio level information is generated in the audio signal processor 107and input to the microprocessor 111. The audio level information is alsoused, for example, for display of an audio level meter on a screen.

On the basis of the audio level information, the microprocessor 111obtains a time at which the audio level exhibits a peak. The audio leveloften reaches a peak at an exciting scene. Thus, a peak of the audiolevel can be considered likely to correspond to an important scene, andtherefore can be considered as a highlight point. Various algorithms maybe employed to detect a peak, and an example will be described below.

During recording of a moving picture, an audio level is obtained. In thecase of a stereo microphone, audio levels for left and right channelscan be obtained, so that, for example, an average of the two values isused as the audio level. Furthermore, in order to alleviate the effectof noise, an average audio level over a predetermined past period (e.g.,3 seconds) is taken. The average value over the predetermined periodwill be referred to herein as a sample. That is, when the recording ofthe moving picture has been finished, samples of audio level atpredetermined intervals have been obtained. Then, a maximum value and aminimum value of the samples are obtained, and normalization isperformed so that the maximum value becomes 10 and the minimum valuebecomes 0.

FIG. 6 shows an example of the normalized data. FIG. 6 is a graph inwhich the horizontal axis represents time and the vertical axisrepresents normalized value of audio level. As shown in FIG. 6, throughnormalization, all the samples take on values in a range of 0 to 10.Then, samples making local maximum values are extracted while the othersamples are discarded. In FIG. 6, normalized values of audio levelforming peaks are indicated by black dots.

Then, samples with audio levels less than a predetermined value (e.g.,8) are also discarded. Then, the remaining samples take on valuesgreater than the predetermined value and also take on local maximumvalues (peaks). Then, when the selected points include temporallyadjacent samples, the sample with a larger value is kept while the othersample is discarded. The samples remaining after these operations areconsidered as highlight points, and a set of recording time and audiolevel is recorded for each of the highlight points.

In the example shown in FIG. 6, first, as candidates of highlightpoints, points having audio levels (normalized values) greater than orequal to the predetermined value (e.g., 8) are selected, namely, pointsP1 to P4. Then, when the selected points (P1 to P4) include anytemporally adjacent samples, a sample with a larger value is kept andthe other sample is discarded. When temporal adjacency is defined as arange of ±5 seconds, the selected point P4 exists within the range oftemporal adjacency of ±5 seconds with respect to the selected point P1having a higher level, so that the selected point P4 is excluded fromthe candidates of highlight points. As a result, only the points P1, P2,and P3 are extracted as highlight points from the data shown in FIG. 6.

The processing sequence of the highlight-point extracting process thatis executed when audio signal level information is input from the audiosignal processor 107 to the microprocessor 111 will be described withreference to a flowchart shown in FIG. 7.

In step S301, the microprocessor 111 obtains audio signal levelinformation from the audio signal processor 107. In step S302, themicroprocessor 111 calculates an average audio level over apredetermined past period (e.g., 3 seconds), and stores the averageaudio level on the memory 120 together with a recording time. Thesesteps are repeated until it is determined in step S303 that videorecording has been finished. When it is determined in step S303 thatvideo recording has been finished, the process proceeds to step S304.

In step S304, normalization is executed on the basis of the temporalaudio level data stored in the memory 120, thereby obtaining, forexample, the relationship between time and normalized audio level datashown in FIG. 6. Then, highlight points are extracted in step S305. Morespecifically,

(a) Points with audio levels (normalized values) greater than or equalto a predetermined value (e.g., 8) are selected.

(b) When the selected points include temporally adjacent samples, asample having a larger value is kept while the other sample isdiscarded.

The points remaining after these steps (a) and (b) are selected ashighlight points. Then, in step S306, sets of time information andnormalized audio level (peak value) for the individual highlight pointsextracted are recorded, for example, on the recording medium 112 inassociation with captured data as highlight point information associatedwith the captured data.

A specific example of the highlight-point extracting process based onaudio level information and a specific example of data of highlightpoint information based on audio level information will be describedwith reference to FIGS. 8A and 8B. FIG. 8A is a diagram showing ahistory of audio level information in relation to time elapsed from thebeginning (Start) to end (Stop) of taking of a moving picture. FIG. 8Bis a diagram showing an example of data of highlight point informationassociated with captured video data, which is recorded on the recordingmedium 112 when the audio levels shown in FIG. 8A are obtained.

In the example shown in FIG. 8A, three highlight points are obtainedfrom the audio level transition data through the highlight-pointextracting process described above. As a result, recording times [2:14],[3:30], and [4:07] are selected as highlight points, as shown in FIG.8B. Thus, highlight point information including three entries as shownin FIG. 8B, i.e., sets of time information and normalized audio level(peak value) for the individual highlight points, are recorded on therecording medium 112 in association with captured data as highlightpoint information associated with the captured data.

(4) Position Information Supplied from the GPS Module 110

Next, a highlight-point extracting process that is executed by themicroprocessor 111 when position information is input from the GPSmodule 110 to the microprocessor 111 will be described with reference toFIG. 9 and FIGS. 10A to 10C. In this process, the position of theimaging device is calculated on the basis of information input from theGPS module 110 during imaging, and the microprocessor 111 determines andrecords highlight points on the basis of changes in the position.

During taking of a moving picture, the microprocessor 111 receivesinformation of GPS satellites from the GPS module 110, so that themicroprocessor 111 can obtain the latitude and longitude of the currentposition in real time. During taking of the moving picture, themicroprocessor 111 calculates the distance of movement or the like ofthe imaging device on the basis of the GPS data.

Since latitude and longitude information calculated on the basis of theGPS data includes a certain degree of error, it is assumed that thedevice has moved when the position has changed by a distance greaterthan or equal to a predetermined distance from the position previouslyobserved. When a considerable amount of movement has occurred duringimaging, it is presumed that a scene captured immediately after themovement is an important scene and therefore corresponds to a highlightpoint. Based on this estimation, the microprocessor 111 calculates adistance of movement on the basis of change in latitude and longitudeaccording to the information input from the GPS module 110. When amovement with a distance greater than or equal to the predetermineddistance has occurred, the microprocessor 111 records a recording timeand an amount of movement associated with the movement on the memory120. When recording of the moving picture has been finished, all thehighlight points stored on the memory 120, or a predetermined number ofhighlight points among the highlight points stored on the memory 120,are recorded on the recording medium 112 together with moving-picturedata.

Now, processing executed in the individual steps of the processing flowshown in FIG. 9 will be described. In step S401, the microprocessor 111receives input of information from the GPS module 110, i.e., informationrepresenting a latitude and longitude as position information of theimaging device. The position information is repeatedly input atpredetermined intervals, and the position information is stored on thememory 120 in association with times of input of the information.

In step S402, the GPS module 110 calculates the difference between theinformation input from the GPS module 110 and position information thathas already been stored on the memory 120, thereby determining whether amovement with a distance greater than or equal to a predeterminedthreshold distance has occurred. When a movement with a distance greaterthan or equal to the threshold distance has occurred, the microprocessor111 determines that a highlight point has been detected. Then, theprocess proceeds to step S411, in which the microprocessor 111 stores adata set of (current recording time, amount of position change) on thememory 120.

When it is determined in step S402 that a movement with a distancegreater than or equal to the predetermined threshold distance has notoccurred, or after the data set of (current recording time, amount ofposition change) has been stored on the memory 120 in step S411, theprocess proceeds to step S404, in which the microprocessor 111determines whether video recording has been finished. When videorecording has not been finished, the process returns to step S401 and isrepeated therefrom. Through this repetition, during image recording,sets of time of occurrence of a movement with a distance greater than orequal to the threshold and amount of position change, i.e., sets of(current recording time, amount of position change), are sequentiallystored on the memory 120.

When it is detected in step S403 that video recording has been finished,in step S404, the sets of (current recording time, amount of positionchange) stored on the memory 120 are recorded, for example, on therecording medium 112 in association with captured data as highlightpoint information associated with the captured data.

Now, a specific example of the highlight-point extracting process basedon GPS information and a specific example of data of highlight pointinformation based on GPS information will be described with reference toFIGS. 10A to 10C. FIG. 10A is a diagram showing times of occurrence ofmovements with distances greater than or equal to the threshold,measured in terms of time elapsed from the beginning (Start) to end(Stop) of taking of a moving picture, i.e., a diagram showing highlightpoints. FIG. 10B is a diagram showing the status of position changecorresponding to the data shown in FIG. 10A.

Imaging is started at an imaging start point P1 shown in FIG. 10B, andthe imaging device starts movement with the user taking a picture by theimaging device. When the imaging device has reached a point P2, it isdetermined that a movement with a distance greater than or equal to thethreshold has occurred, so that the current point in time is determinedas a first highlight point. The time associated with this point is 1minutes and 35 seconds from the beginning of imaging, and this time andthe amount of movement are recorded as the first entry of highlightpoint information shown in FIG. 10C. Furthermore, when the imagingdevice has moved from the point P2 to a point P3, it is determined thata movement with a distance greater than or equal to the threshold hasoccurred, so that the current point in time is determined as a secondhighlight point. The time associated with this point is 2 minutes and 11seconds after the beginning of imaging, and this time and the amount ofmovement are recorded as the second entry of highlight information shownin FIG. 10C.

(5) Date and Time Information Supplied from the Clock 131

Next, a highlight-point extracting process that is executed by themicroprocessor 111 when date and time information is input from theclock 131 to the microprocessor 111 will be described with reference toFIG. 11 and FIGS. 12A and 12B. In this process, during imaging, date andtime information is input from the clock 131 to the microprocessor 111,and the microprocessor 111 detects and records highlight points on thebasis of the information representing the current date and time.

During taking of a moving picture, the microprocessor 111 receives inputof information representing a current date and time from the clock 131.The current time is measured based on a time specified in advance by theuser or set according to GPS satellite information received from the GPSmodule 110. The microprocessor 111 monitors the clock 131 during takingof a picture, so that the microprocessor 111 can obtain recording timeinformation of a point in time when the date changes, a point in timewhen the year changes, and so forth.

At a moment when the date changes, the year changes, or the like, it ispossible to presume that a certain important event is being held, sothat the moment can be considered as a highlight point. In this example,recording times at points when the date changes or the year changesduring imaging are considered as highlight points.

Now, the sequence of a highlight-point detecting and recording processexecuted by the microprocessor 111 on the basis of date and timeinformation supplied from the clock 131 will be described with referenceto a flowchart shown in FIG. 11. In step S501, the microprocessor 111receives input of date and time information from the clock 131. In stepS502, the microprocessor 111 determines whether the date and timeinformation corresponds to a date and time of an event. The date andtime of an event is determined on the basis of, for example, data thatis set in advance by a user. Furthermore, points of date changes, yearchanges, and so forth may be defined as event dates and times bydefault.

When it is determined in step S502 that the date and time informationcorresponds to an event date and time, it is determined that a highlightpoint has been detected. Then, in step S511, a data set of (currentrecording time, event type) is stored on the memory 120. The event typeindicates, for example, a date change or a year change.

When it is determined in step S502 that the date and time informationdoes not correspond to any event date and time, or after the data set of(current recording time, event type) has been stored on the memory 120,in step S503, the microprocessor 111 determines whether video recordinghas been finished. When video recording has not been finished, theprocess returns to step S501 and is repeated therefrom. Through thisrepetition, sets of time of occurrence of an event during imagerecording and information representing an event type, i.e., sets of(current recording time, event type), are sequentially stored on thememory 120.

When it is detected in step S503 that video recording has been finished,in step S504, the microprocessor 111 records the sets of (currentrecording time, event type) stored on the memory 120, for example, onthe recording medium 112 in association with captured data as highlightpoint information associated with the captured data.

Now, a specific example of the highlight-point extracting processedbased on clock information and a specific example of highlight pointinformation based on clock information will be described with referenceto FIGS. 12A and 12B. FIG. 12A is a diagram showing points in time whereoccurrences of evens are detected on the basis of clock informationalong time elapsed from the beginning (Start) to end (Stop) of taking ofa moving picture. FIG. 12B is a diagram showing an example of data ofhighlight point information recorded on the recording medium 112 inassociation with captured data shown in FIG. 12A.

In the example shown in FIG. 12A, one event occurrence time, e.g., atime of year change, is included between the beginning and end ofimaging, and this point is recorded as a highlight point. The datarecorded corresponds to entry data of highlight point information shownin FIG. 12B. More specifically, a data set of a recording time from thebeginning of imaging corresponding to the time of the year change, i.e.,[1:40], and event type, i.e., [year change], is recorded as highlightpoint information.

(6) Still-Picture Taking Information Associated with User's Operationsof the Input Devices 108

Next, a highlight-point extracting process that is executed by themicroprocessor 111 when still-picture taking information associated witha user's operation of the input devices 108 is input to themicroprocessor 111 will be described with reference to FIG. 13 and FIGS.14A and 14B.

FIG. 13 is a flowchart showing the processing sequence of ahighlight-point extracting process that is executed by themicroprocessor 111 during imaging when still-picture taking informationassociated with a user's operation of the input devices 108 is input tothe microprocessor 111.

The user can take a still picture by pressing a still picture key (e.g.,a shutter key), which is one of the input devices 108, while taking amoving picture. The user's operation is reported from the input devices108 to the microprocessor 111, and the microprocessor 111 controls thecamera to take a still picture. When the user takes a still picturewhile taking a moving picture, it is presumed that the user considersthe scene as important, so that the still picture can be considered ascorresponding to a highlight point. In this example, a recording timecorresponding to a time of taking a still picture is recorded as ahighlight point.

Now, processing executed in the individual steps of the processing flowshown in FIG. 13 will be described. When the microprocessor 111 detectsa shutter key operation for taking a still picture in step S601, theprocess proceeds to step S611, in which a still picture is taken. Then,in step S612, information representing the time of taking the stillpicture is stored on the memory 120.

Then, in step S602, it is detected whether video recording has beenfinished. When video recording has not been finished, the processreturns to step S601 and is repeated therefrom. Through this repetition,data of times of taking still pictures during video recording issequentially stored on the memory 120.

When it is detected in step S602 that video recording has been finished,in step S603, the data (recording times of taking still pictures) storedon the memory 120 is recorded on the recording medium 112 in associationwith captured data as highlight information associated with the captureddata.

Now, a specific example of the highlight-point extracting process basedon still-picture taking information and a specific example of data ofhighlight point information based on still-picture taking informationwill be described with reference to FIGS. 14A and 14B. FIG. 14A shows ahistory of captured still pictures along time elapsed from the beginning(Start) to end (Stop) of taking of a moving picture. FIG. 14B shows anexample of data of highlight points information associated with capturedvideo data, recorded on the recording medium 112 when still pictures aretaken as shown in FIG. 14A.

In the example shown in FIG. 14A, still pictures are taken twice betweenthe beginning and end of imaging. The first still picture is taken at 2minutes and 22 seconds after the beginning of imaging. The time oftaking the still picture is recorded as a highlight point. The datarecorded corresponds to the first entry data of highlight pointinformation shown in FIG. 14B, and the time of taking the still picture,i.e., [2:22], is recorded as a first piece of highlight pointinformation. Similarly, second and third pieces of highlight pointinformation are sequentially recorded.

Overall Processing Sequence

As described above, the imaging device according to this embodimentextracts highlight points on the basis of the following pieces ofinformation:

(1) Zoom operation information associated with user's operations of theinput devices 108

(2) Sensor detection information supplied from the acceleration sensor109

(3) Audio signal level information supplied from the audio signalprocessor 107

(4) Position information supplied from the GPS module 110

(5) Date and time information supplied from the clock 131

(6) Still-picture taking information associated with user's operationsof the input devices 108

These processes can be executed simultaneously on a single movingpicture. In this case, highlight point information is generated on thebasis of these various types of information and is recorded inassociation with captured video data. The highlight point informationrecorded is, for example, data shown in FIGS. 15A to 15F.

FIGS. 15A to 15F show highlight point information extracted an recordedby the microprocessor 111 on the basis of the following pieces ofinformation (1) to (6), respectively:

(1) Zoom operation information associated with user's operations of theinput devices 108

(2) Sensor detection information supplied from the acceleration sensor109

(3) Audio signal level information supplied from the audio signalprocessor 107

(4) Position information supplied from the GPS module 110

(5) Date and time information supplied from the clock 131

(6) Still-picture taking information associated with user's operationsof the input devices 108

In the imaging device according to this embodiment, pieces of highlightpoint information generated on the basis of these pieces of informationis recorded on a recording medium, together with priority levelsassigned to the individual pieces of highlight point information inaccordance with the types of input information used to generate thepieces of highlight point information. The priority levels are used toselectively play back only highlight scenes.

Now, a procedure of selectively playing back only highlight scenes ofcaptured video data using highlight point information recorded inassociation with the captured video data will be described. The imagingdevice according to this embodiment is capable of playing back acaptured moving picture on the display 127, and selectively playing backscenes corresponding to highlight points using highlight pointinformation recorded in association with the captured moving picture.

When playing back highlight scenes, for example, a user selects a singlemoving picture and play back only highlight scenes thereof, or highlightscenes of a certain number of moving pictures are played back in theform of a slide show.

When playing back highlight scenes, it is possible to select and playback all the large number of highlight scenes corresponding to thehighlight points shown in FIGS. 15A to 15F. For example, on the basis ofthe recording times included in the respective pieces of highlightinformation shown in FIGS. 15A to 15F, it is possible to playbackhighlight scenes using the recording times as indices, in accordancewith setting such as preceding 10 seconds and succeeding 10 seconds orpreceding 5 seconds and succeeding 20 seconds with respect to thecaptured picture frame corresponding to each recording time.

Instead of selecting and playing back all the scenes corresponding tothe large number of highlight points shown in FIGS. 15A to 15F, it ispossible to select highlight points with higher priority levels on thebasis of the priority levels assigned to the highlight points and toplay back only the scenes corresponding to the selected highlightpoints.

Now, the priority levels of highlight points will be described. Asdescribed earlier, highlight points are extracted on the basis ofvarious types of information, such as zoom operations, pan or tiltoperations, peaks of audio level, changes in position based on GPSinformation, changes in date or year based on time information, andinformation regarding still pictures captured while taking a movingpicture. Priority levels are assigned to these highlight pointsaccording to the types of information used to extract the highlightpoints. The priority levels can be assigned arbitrarily by the user.

For example, when the user considers that highlight points associatedwith zoom operations, peaks of audio level, and still pictures capturedwhile taking a moving picture are more likely to correspond to importantscenes while highlight points associated with pan or tilt operations,changes in position based on GPS information, and changes in date oryear based on time information are less likely to correspond toimportant scenes, the user assigns priority levels in the followingmanner:

Priority level 1: Highlight points extracted on the basis of zoomoperations, peaks of audio level, and captured still pictures

Priority level 2: Highlight points extracted on the basis of GPSinformation and events such as date or year changes

Priority level 3: Highlight points extracted on the basis of pan or tiltoperations

After setting the priority levels as described above, the number ofhighlight points to be selected for selective playback of highlightpoints is determined. In order to play back only highlight scenes of amoving picture, first, a length of highlight playback is determined. Thelength may be determined in advance as a fixed value (e.g., 20 seconds)or as a fixed ratio to the length of the entire moving picture (e.g.,one fifth of the length of the moving picture), or set in advance by theuser.

Then, the length of each highlight scene to play back is determined. Thelength may be determined in advance as a fixed value (e.g., 5 seconds),or the length may be determined on each occasion of playing back amoving picture in the form of a slide show to give certain presentationeffect. When BGM is played back in a slide show, the length may bedetermined for each bar of the music or set by the user in advance. Inthis manner, the entire length and the length per scene for highlightplayback can be determined. These two values can be selectedappropriately in accordance with a desired manner of playback. Thenumber of highlight points to be selected can be calculated by dividingthe entire length of highlight playback by the length per scene. Whenthe length per scene is variable as in a case where the length isdetermined in accordance with BGM, the number of highlight points to beselected corresponds to the number of scenes with which the total lengthof the scenes amounts to the length of the BGM.

Next, a procedure of selecting highlight points will be described withreference to a flowchart shown in FIG. 16. In the highlight pointselecting process, before playing back highlight points, themicroprocessor 111 selects highlight points with higher priority levelusing highlight point information associated with captured video data.

First, in step S701, a set of highlight points having a highest prioritylevel assigned thereto are selected. In the case of the example ofpriority level setting described above, a set of the following highlightpoints are obtained: Priority level 1: Highlight points extracted on thebasis of zoom operations, peaks of audio level, and captured stillpictures

Then, in step S702, it is checked whether the set is empty. When the setis not empty, the process proceeds to step S703, one highlight point isselected from the set. In this selection, when a plurality of highlightpoints having the same priority level assigned thereto exist, forexample, one highlight point may be selected at random. Alternatively,for example, one highlight point may be selected on the basis of anindex representing a degree of change, e.g., an amount of zoom change inthe case of a zoom operation, an amount of movement in the case of a panor tilt operation, a value of audio level in the case of audio level, oran amount of position change in the case of latitudinal or longitudinalchange based on GPS information.

In step S704, it is checked whether the number of highlight pointsobtained has reached a predetermined upper limit. When the number ofhighlight points has not reached the upper limit, the process returns tostep S702 and is repeated therefrom. When the set of highlight pointswith the highest priority level has become empty, the process proceedsto step S711, in which it is checked whether a set of highlight pointswith a next highest priority level exist. When a set of highlight pointswith a next highest priority level exists, in step S712, the set ofhighlight points having the next highest priority level are selected. Inthe case of the example of priority level setting described above, thefollowing set of highlight points are obtained: Priority level 2:Highlight points extracted on the basis of GPS information or eventssuch as date changes or year changes

Then, in step S703, one highlight point is selected from the selectedset. These steps are repeated until the number of highlight pointsselected sequentially from those with higher priority levels reaches apredetermined number or until all the highlight points are extracted.

When it is determined in step S711 that no set of highlight points to beselected exists, in step S713, the captured picture is divided intosegments (e.g., 10 segments) with equal lengths, and the resultingpoints are considered as a set of highlight points. Then, the processreturns to step S703, in which a highlight point is selected on thebasis of the new set. The selection is executed according to apredetermined rule; for example, highlight points formed by dividing themoving picture equally are selected in order from the beginning.

Finally, the highlight point selecting process is ended when it isdetermined in step S704 that the number of highlight points has reacheda predetermined number. By repeating selection in order of prioritylevel as described above, a predetermined number of highlight points areselected.

Next, a procedure of playing back highlights of a moving picture on thebasis of selected highlight points will be described. First, selectedhighlight points are arranged in order of imaging time. Then, a scene ofmoving picture associated with each highlight point is played back forthe length per scene, starting from a little before the highlight point.For example, when the length per scene is 5 seconds, the scene of movingpicture is played back for 5 seconds starting from 1 second before thehighlight point. This rule is applied sequentially to all the highlightpoints, whereby only highlight scenes of the moving picture are playedback. FIG. 17 shows an example of selective playback of highlightscenes.

FIG. 17 shows an example where three highlight points are selected andplayed back from captured picture data. Highlight scenes 301 to 303 aremoving-picture data including highlight points selected by the processdescribed above. For example, setting is made so that a playback scenecovers a range from 1 second before a highlight point to 5 seconds afterthe highlight point. In the example shown in FIG. 17, only threehighlight scenes are played back. For example, setting may be made sothat an effect of scene change, such as cross fading, is insertedbetween highlight points.

Film-Roll Displaying Process

Next, a film-roll displaying process that is executed using thehighlight point information described above will be described. First, anoverview of film-roll display will be described. An example of thefilm-roll displaying process will be described with reference to FIGS.18 and 19.

FIG. 18 shows an example where a film-roll display area 502 issuperposed on a moving-picture playback screen 501 for playing back acaptured moving picture on a display. In the film-roll display area 502,a plurality of representative still pictures selected from themoving-picture data file being played back in the moving-pictureplayback screen 501 are displayed. By arranging a plurality of framepictures (thumbnail pictures) in the moving picture in a temporal orderat certain appropriate intervals, it is possible to recognize thecontent of the moving picture and to facilitate selection of a playbackposition with reference to scenes preceding and succeeding the scenebeing played back.

FIG. 19 shows an example where representative still pictures of aplurality of captured moving-picture data files are displayed in theform of a film roll. In the example shown in FIG. 19, representativestill pictures for four moving-picture data files are displayed in atemporal order in film-roll display areas 601 to 604, respectively.

When a list of captured moving pictures stored on a recording mediumwithin the device is displayed, it is difficult to recognize the contentof each moving picture by displaying one representative picture (e.g., athumbnail of the beginning frame) for the moving picture. In contrast,by displaying a film roll for each moving picture as shown in FIG. 19,it becomes easier to recognize the content of the moving picture, andthis facilitates selection of which moving picture to play back.

In an ordinary film-roll display, pictures of individual frames of themoving picture (thumbnail pictures) are used. The thumbnail pictures canbe obtained by decoding specific frame pictures selected from the movingpicture. Alternatively, frame pictures at predetermined intervals aresaved as still pictures (thumbnail pictures) while taking a movingpicture so that the still pictures can be used for film-roll display.When thumbnail pictures of arbitrary frames (or frames at predeterminedintervals) can be obtained, by arranging the thumbnail images in adisplay memory and displaying the thumbnail images, the film-rolldisplay shown in FIG. 18 or FIG. 19 can be achieved.

Next, a film-roll displaying process using the highlight pointinformation described earlier will be described. When captured picturedata is displayed in the form of a film roll, in order to simultaneouslydisplay still-picture frame data selected from picture data from thebeginning to end of a moving-picture data file, frames are extracted atsuch intervals that the entire length of the moving picture is coveredand thumbnail images of the extracted frames are displayed.

However, particularly in a device having a limited size of liquidcrystal screen, such as a digital camera, the number of thumbnailpictures that can be arranged simultaneously is limited to severalpictures (e.g., 5 pictures). For example, in order to cover the lengthof a 5-minute moving picture by five thumbnail pictures, thumbnailpictures are arranged at 1-minute intervals. However, as the intervalsof thumbnail pictures become longer, it becomes more difficult torecognize the content of the moving picture. Furthermore, thumbnailpictures of frames extracted at regular intervals might not containsufficient information for identifying scenes (e.g., a picture in themiddle of a scene change by a camera pan). In that case, even with thefilm-roll display, it is not possible to determine what the scene islike. In contrast, when executing film-roll display, by selecting framescorresponding to highlight points that are likely to represent importantscenes and displaying thumbnail pictures for the frames, selectedimportant scenes are used for the film-roll display. Thus, the film rolldisplay areas can be used more effectively even with a small number offrames displayed.

As described earlier, highlight points are stored on the recordingmedium 112 as data including identification information representingpositions of important scenes. The imaging device according to thisembodiment selects pictures to be displayed in film-roll display usingthe highlight point information.

Now, a sequence of a film-roll displaying process executed by theimaging device according to this embodiment will be described withreference to a flowchart shown in FIG. 20. FIG. 20 shows a flowchart ofa process executed by the microprocessor 111 as a data processor, inwhich a film-roll display of captured moving pictures stored on therecording medium 112 is executed using highlight point informationassociated with the captured moving pictures.

Before executing steps in the flowchart shown in FIG. 20, first, amoving-picture data file that is to be subjected to a film-roll displayis determined, and the number of thumbnail pictures to be displayed in ascreen is determined. By dividing the length (recording time) of themoving picture by the number of still pictures used in the film-rolldisplay, the display intervals (time intervals) of still-picture framesused in the film-roll display can be obtained. The number of thumbnailsof still-picture frames used for the film-roll display is determined inaccordance with the layout of the display screen.

When still pictures used for the film-roll display are notsimultaneously displayed in the screen but a function such as scrollingoperation is used, the display intervals of frames may be determinedarbitrarily. For example, it is possible to display thumbnail picturesof frames at certain predetermined intervals, such as 5-second intervalsor 10-second intervals. In this case, since it is not possible todisplay all the thumbnail pictures simultaneously, film-roll display isexecuted while changing pictures by scrolling operations or the like.

Now, processing executed in the individual steps of the flowchart shownin FIG. 20 will be described. In step S801, Δx representing a frameinterval (recording time interval) for film-roll display (Δx>0) and x1representing a reference time (recording time) of a frame that isdisplayed at the beginning of a film roll are determined.

That is, the recording times of the frames (still pictures) that are tobe displayed in the film-roll display can be represented as follows:

First frame: Reference time x(1)=x(1)

Second frame: Reference time x(2)=x(1)+Δx

Third frame: Reference time x(3)=x(1)+2Δx

i-th frame: Reference time x(i)=x(1)+(i−1)Δx

where i is a parameter representing the number of a frame that is usedin a film-roll display, as counted from the beginning.

In step S802, the parameter i is initialized so that i=1. Then, in stepS803, in order to determine a thumbnail picture as a frame picture to bedisplayed in an i-th display area of the film-roll display, highlightpoint information recorded in association with the picture datasubjected to the film-roll display, e.g., the various types of highlightpoint information described earlier with reference to FIGS. 15A to 15F,is searched for a time tj of a highlight point that is closest to thereference time x(i).

For example, the various types of highlight point information describedearlier with reference to FIGS. 15A to 15F are contained in a singledata file, and the recording times tp (p=1, . . . , n) are sorted inorder of time, and a highlight point tj that is closest to the referencetime x(i) can be determined efficiently by using binary search.

Then, in step S804, it is checked whether the highlight point tj closestto the reference point x(i), determined from the highlight pointinformation, satisfies expression (1) below:x−(Δx/2)≦tj<x+(Δx/2)  (1)

Expression (1) is used to check whether the highlight point tj closestto the reference point x(1), obtained from the highlight pointinformation, is closer to x(i) than the frames preceding and succeedingthe reference time x(i) of the i-th picture frame used for the film-rolldisplay, i.e., the reference time x(i−1) of the (i−1)-th picture frameand the reference time x(i+1) of the (i+1)-th picture frame.

When expression (1) is satisfied, the process proceeds to step S811, inwhich it is determined that the i-th picture frame of the subject of thefilm-roll display can be replaced with the frame at the time of thehighlight point tj, so that the frame at the time of the highlight pointtj is used as the i-th picture frame of the subject of the film-rolldisplay. As described above, the picture can be displayed by decodingthe moving picture or by using a saved still picture (thumbnail image)of the frame.

When it is determined in step S804 that expression (1) is not satisfied,x−(Δx/2)≦tj<x+(Δx/2)  (1)the process proceeds to step S805. In step S805, considering that thei-th picture frame of the subject of the film-roll display is notreplaceable with any highlight point, the frame picture at time x(i) isdisplayed as the i-th picture frame of the subject of the film-rolldisplay.

Then, in step S806, it is checked whether the number of picturesselected for the film-roll display has reached a predetermined number.When the number of pictures has not reached the predetermined number, instep S807, the parameter i is incremented by 1. Then, in step S808, thereference time (recording time) of the i-th display frame is calculatedaccording to:x(i)=x(i−1)+ΔxThen, the process returns to step S803 and is repeated therefrom.

This process is repeated until the number of pictures selected for thefilm-roll display reaches the predetermined number, and the process isended when it is determined in step S806 that the number of picturesselected for the film-roll displayed has reached the predeterminednumber.

Through this process, many of the pictures that are to be used in thefilm-roll display are replaced with pictures corresponding to highlightpoints selected as important scenes. Thus, the user can efficientlyrecognize the content of the moving picture and to select scenes to playback. Furthermore, since the pictures selected for the film-roll displayare arranged in a temporal order, temporal inconsistencies do not arise.

Next, a procedure of changing the range of film-roll display inaccordance with user's operations will be described. As describedearlier, the time intervals of frames used for film-roll display may bedetermined arbitrarily. However, the minimum value of the frameintervals is determined according to a minimum time in which a framepicture can be obtained.

For example, when captured video data is recorded as MPEG data, in adevice in which only I pictures of GOPs (Groups of pictures) defined byMPEG are selected as frame pictures used for film-roll display, theminimum value of the intervals of pictures used for film-roll displaycorresponds to a period of a GOP. The minimum value will be denoted asdmin. The maximum value of a frame period is determined according to thelength of the moving picture. For example, the maximum value is obtainedby dividing the length of the moving picture by the number of areas fordisplaying frame pictures. The maximum time will be denoted as dmax.

The frame intervals of pictures displayed in the film-roll display cantake on the following values:

dmin, dmin×2, dmin×3, . . . , dmin×k, dmax

where k is a natural number and dmin×k<dmax≦dmin×(k+1).

When a film-roll display according to the procedure described above hasbeen going on with the frame time interval Δx=d, and the film-rolldisplay is to be changed to a film roll display with Δx=dnew, forexample, a film-roll display is executed with the time x1 at thebeginning of the film roll maintained as it is and the times x2, . . . ,xn calculated by the above procedure using Δx=dnew.

By the procedure described above, a film-roll display can be presentedwith various time intervals. Thus, for example, when the user wishes toview a detailed film-roll display at relatively short time intervals, afilm roll display with relatively short time intervals is presentedaccordingly. Conversely, when the user wishes to grasp the content ofthe whole moving picture at rather rough time intervals, a film-rolldisplay with relatively long time intervals is presented accordingly.

Next, a procedure of playing back a moving picture from a position of aframe in a film-roll display will be described. When a film-roll displayis presented in the manner described above, many frame picturescorresponding to highlight points that are likely to represent importantscenes exist in the frames of the film-roll display. When the userwishes to play back a scene corresponding to a highlight point withreference to the film-roll display, the user selects the correspondingpicture in the film-roll display. The user selects the picture using aninput device, for example, by directly selecting the picture using atouch panel, or by placing a cursor on a frame in the film-roll display,moving the cursor using a button for horizontal movement, and pressing abutton for starting playback with the cursor placed on a frame ofinterest. Information representing the selection by the user is input tothe microprocessor 111. Then, the microprocessor 111 obtains thehighlight point (recording time) corresponding to the selected framepicture from highlight point information, and starts playing back themoving picture from the recording time. Alternatively, themicroprocessor 111 starts playback from a point a little before therecording time of the highlight point, thereby playing back a scene inthe proximity of the highlight point.

The present invention has been described in detail in the context ofspecific embodiments. However, obviously, it is possible for thoseskilled in the art to make modifications or alternatives withoutdeparting from the spirit of the present invention. That is, the presentinvention has been disclosed by way of examples, and the presentinvention should not be construed as limited to the embodiments. Thescope of the present invention should be determined on the basis of theclaims.

The series of processes described herein can be executed by hardware, bysoftware, or by combination of hardware and software. When the series ofprocesses is executed by software, a program defining the processingsequences is executed by installing it on a memory of a computerembedded in special hardware or on a general-purpose computer that iscapable of executing various processes.

For example, the program may be recorded in a recording medium such as ahard disc or a read-only memory (ROM). Alternatively, the program may bestored (recorded) temporarily or permanently on a removable recordingmedium such as a flexible disc, a compact disc read-only memory(CD-ROM), a magneto-optical (MO) disc, a digital versatile disc (DVD), amagnetic disc, or a semiconductor memory. Such a removable medium can beprovided in the form of what is called package software.

Instead of installing the program from a removable recording medium to acomputer, the program may be transferred by wireless from a downloadsite to a computer, or transferred by wire to a computer via a networksuch as a local area network (LAN) or the Internet, so that the computercan receive the program transferred and install the program on aninternal recording medium such as a hard disc.

The various processes described herein need not necessarily be executedsequentially in the orders described, and may be executed in parallel orindividually as needed or in accordance with the processing ability ofan apparatus that executes the processes. A system in this specificationrefers to a logical combination of a plurality of apparatuses,regardless of whether the constituent apparatuses are disposed withinthe same case.

1. An information processing apparatus comprising: a data processorconfigured to execute processing for selecting a plurality of framepictures from moving-picture data and for displaying an array of theselected frame pictures, wherein, in selecting frame pictures to bedisplayed, the data processor preferentially selects frame picturescorresponding to highlight points as frame pictures to be displayed,using highlight point information that allows identification of pictureportions corresponding to important scenes, the highlight pointinformation being recorded in association with the moving-picture data.2. The information processing apparatus according to claim 1, wherein,in selecting frame pictures to be displayed, the data processorcalculates reference times corresponding to a plurality of displayframes that are set along a temporal sequence, and for each of thereference times, the data processor preferentially selects a framepicture corresponding to a highlight point having a recording timenearest to the reference time as a frame picture to be displayed.
 3. Theinformation processing apparatus according to claim 1, wherein, inselecting frame pictures to be displayed, the data processor calculatesreference times corresponding to a plurality of display frames that areset along a temporal sequence, and for each of the reference times, thedata processor preferentially selects a frame picture corresponding to ahighlight point having a recording time nearest to the reference time asa frame picture to be displayed, and when no frame picture correspondingto the highlight point exists in a predetermined threshold time range,the data processor selects a frame picture corresponding to thereference time as a frame picture to be displayed.
 4. The informationprocessing apparatus according to claim 1, wherein the data processorobtains recording times of important scenes from highlight pointinformation including the recording times of the important scenes, andfor each of the recording times of the important scenes, the dataprocessor selects a frame picture corresponding to a highlight pointhaving the recording time of the important scene as a frame picture tobe displayed when the following conditional expression is satisfied:x−(Δx/2)≦tj<x+(Δx/2) where tj denotes the recording time of theimportant scene, x denotes a reference time corresponding to a displayframe associated with the recording time of the important scene among aplurality of display frames that are set along a temporal sequence, andΔx denotes a time interval of the display frames.
 5. An imaging devicecomprising: a recording data processor configured to execute processingfor selecting picture portions corresponding to important scenes fromcaptured picture data on the basis of input information that is inputduring a picture taking period, for generating highlight pointinformation that allows identification of the selected picture portionscorresponding to the important scenes, and for recording the highlightpoint information on a recording medium as information associated withthe captured picture data; and a playback data processor configured toexecute processing for selecting a plurality of frame pictures frommoving-picture data and for displaying an array of the selected framepictures; wherein, in selecting frame pictures to be displayed, theplayback data processor preferentially selects frame picturescorresponding to highlight points as frame pictures to be displayed,using the highlight point information recorded in association with themoving-picture data.
 6. The imaging device according to claim 5, whereinthe recording data processor receives, as the input information, atleast one of zoom operation information, detection information of anacceleration sensor, audio signal level information, GPS information,clock information, and still-picture taking information, identifies thepicture portions corresponding to the important scenes on the basis ofthe input information, and generates the highlight point information. 7.An information processing method for executing processing for displayinga picture by an information processing apparatus, the informationprocessing method comprising the step of: executing data processing forselecting a plurality of frame pictures from moving-picture data and fordisplaying an array of the selected frame pictures, the data processingbeing executed by a data processor, wherein, in selecting frame picturesto be displayed, frame pictures corresponding to highlight points arepreferentially selected as frame pictures to be displayed, usinghighlight point information that allows identification of pictureportions corresponding to important scenes, the highlight pointinformation being recorded in association with the moving-picture data.8. The information processing method according to claim 7, wherein, inselecting frame pictures to be displayed, reference times correspondingto a plurality of display frames that are set along a temporal sequenceare calculated, and for each of the reference times, a frame picturecorresponding to a highlight point having a recording time nearest tothe reference time is preferentially selected as a frame picture to bedisplayed.
 9. The information processing method according to claim 7,wherein, in selecting frame pictures to be displayed, reference timescorresponding to a plurality of display frames that are set along atemporal sequence are calculated, and for each of the reference times, aframe picture corresponding to a highlight point having a recording timenearest to the reference time is preferentially selected as a framepicture to be displayed, and when no frame picture corresponding to thehighlight point exists in a predetermined threshold time range, a framepicture corresponding to the reference time is selected as a framepicture to be displayed.
 10. The information processing method accordingto claim 7, wherein recording times of important scenes are obtainedfrom highlight point information including the recording times of theimportant scenes, and for each of the recording times of the importantscenes, a frame picture corresponding to a highlight point having therecording time of the important scene is selected as a frame picture tobe displayed when the following conditional expression is satisfied:x−(Δx/2)≦tj<x+(Δx/2) where tj denotes the recording time of theimportant scene, x denotes a reference time corresponding to a displayframe associated with the recording time of the important scene among aplurality of display frames that are set along a temporal sequence, andΔx denotes a time interval of the display frames.
 11. An informationprocessing method executed by an imaging device, the informationprocessing method comprising the steps of: executing recording dataprocessing for selecting picture portions corresponding to importantscenes from captured picture data on the basis of input information thatis input during a picture taking period, for generating highlight pointinformation that allows identification of the selected picture portionscorresponding to the important scenes, and for recording the highlightpoint information on a recording medium as information associated withthe captured picture data, the recording data processing being executedby a recording data processor; and executing playback data processingfor selecting a plurality of frame pictures from moving-picture data andfor displaying an array of the selected frame pictures, the playbackdata processing being executed by a playback data processor; wherein, inselecting frame pictures to be displayed, frame pictures correspondingto highlight points are preferentially selected as frame pictures to bedisplayed, using the highlight point information that allowsidentification of the picture portions corresponding to the importantscenes, the highlight point information being recorded in associationwith the moving-picture data.
 12. The information processing methodaccording to claim 11, wherein, as the input information, at least oneof zoom operation information, detection information of an accelerationsensor, audio signal level information, GPS information, clockinformation, and still-picture taking information is received, thepicture portions corresponding to the important scenes are identified onthe basis of the input information, and the highlight point informationis generated.
 13. A computer program for allowing an informationprocessing apparatus to execute processing for displaying an image, thecomputer program comprising the step of: executing data processing forselecting a plurality of frame pictures from moving-picture data and fordisplaying an array of the selected frame pictures, the data processingbeing executed by a data processor, wherein, in selecting frame picturesto be displayed, frame pictures corresponding to highlight points arepreferentially selected as frame pictures to be displayed, usinghighlight point information that allows identification of pictureportions corresponding to important scenes, the highlight pointinformation being recorded in association with the moving-picture data.14. A computer program for allowing an imaging device to executeinformation processing, the computer program comprising the steps of:executing recording data processing for selecting picture portionscorresponding to important scenes from captured picture data on thebasis of input information that is input during a picture taking period,for generating highlight point information that allows identification ofthe selected picture portions corresponding to the important scenes, andfor recording the highlight point information on a recording medium asinformation associated with the captured picture data, the recordingdata processing being executed by a recording data processor; andexecuting playback data processing for selecting a plurality of framepictures from moving-picture data and displaying an array of theselected frame pictures, the playback data processing being executed bya playback data processor; wherein, in selecting frame pictures to bedisplayed, frame pictures corresponding to highlight points arepreferentially selected as frame pictures to be displayed, using thehighlight point information that allows identification of the pictureportions corresponding to the important scenes, the highlight pointinformation being recorded in association with the moving-picture data.